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Process for Reverse Engineering 

Setup an isolated run-time environment 

Execution and initial analysis 

Deobfuscate compressed or packed code 

Disassembly / Code-level Analysis 

Identify and analyze relevant and interesting 
portions of the program 
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Isolated Analysis Environment 

Setup an Isolated Runtime Environment 

- Virtual machines: VMWare, Xen, KVM, ... 

- Need to protect yourself from malicious code 

- Create a known-good baseline environment 

- Quickly allows backtracking if something bad happens 
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Execution and Initial Analysis 

Goal: Quickly figure out what the program is 
doing without looking at assembly 

Look for: 

- Changes to the file system 

- Changes to the behavior of the system 

• Network traffic 

• Overall performance 

• Ads or changed browser settings 
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Remove Software Armoring 

• Program protections to prevent reverse 
engineering 

• Done via packers - Small encoder/decoder 

• Self-modifying code 

• Lots of research about this 

- OllyBonE, Saffron, Polyunpack, Renovo, Ether, 
Azure 

- My research uses Ether 
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Packing and Encryption 

Self-modifying code 
-Small decoder stub 

- Decompress the main executable 

- Restore imports 

Play "tricks" with the executable 

- OS Loader is inherently lazy (efficient) 

- Hide the imports 

- Obscure relocations 

- Use bogus values for various unimportant fields 
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Software Armoring 

- Compressed, obfuscated, hidden code 
-Virtual machine detection 

- Debugger detection 

- Shifting decode frames 
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Normal PE File 




vbp 

ebp. esp 

esp, iCh ; LpNsg 

ds: inp Get Command Line LOG ; 

rebp+iiui^rvrnj-.j] ; nCmdShow 
pa* ; int 

[ebp+hPreu Instance] ; int 
jebp+h Instance] ; hlnstance 
_FSolInit@16 ; FSollnit(jf ,} 

tax, t'dK 

sh&Kt locretliOlFia 
est 

esi , ds : inp GetMessageu@l 6 

edi 

[ebp*Msg .wParan] , 1 

pdi, edi 

shgrt loc_1001EFE 
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Packed PE File 




14899425(1, 43D2D868h, 7460^863(1, I 
*t58959D6h, flFFF883£4h p BEC8D025Dh, 
83591607(1, 2F0OFC65h T 0r73U3BFDh, 
lA0B69F6h t 67E4»30fih 1 1007eF75h, * 
0A32OEE5*th, 46C223E9h, 9EB753Bh, ( 
2B8B7BC2h, 4FF666GCh, BD10ft38DFh p 
BFF03C01Bh T 59F4BBftFh P OBBFflCS^Sht 
6SBGF?3h t ifOFFF.F9E>h f ODOFF02B3h, S 
0DEBB235E11, 4EC2374h, BCC8itFF23h p 
3F839FCh, 3C418B2WI, BfiDC6C183h P J 
0D8FB187SIT, BFE382B3h P 31C094h p 8E 
1tC825'*Bh t 5391441b, 0P233M71h p I 
/r.lHOBWi, 2Jl7CBB1Eh. 3BBC1&14ll p f 
UC72FB3Bh, BFFDADDBSh p 83B1C283h P 
680DC35Bh, 8F4FBFF8h P BBA*t87343h p 
8063D1Fh, IBSBGZZBh* BE9BBC05H, 2E 
(IE 21 /ii i. an. MEmnunn, 60006020(1, 

5A13A2CBh, 1FBBD73flh l 8C7*i&4969h p 



public 


start 


start 


proc near 


push a 




mou 


esi, offset loc UB6000 


lea 


edi, [esi-Bflflflh] 


push 


edi 


or 


ebp T OFFFFFFFFh 


jmp 


short loc 4S6882 
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Troublesome Protections 

• Virtual Machine Detection 

- Redpill, ocvmdetect, Paul Ferrie's paper 

• Debugger Detection 

- IsDebuggerPresentQ 

- EFLAGS bitmask 

• Timing Attacks 

- Analyze value of RDTSC before and after 

- Really effective 
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Thwarting Protections 

Two methods for circumvention 



1. Know about all the protections before hand and 
disable them 



2. Make yourself "invisible" 
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Virtual Machine Monitoring 

Soft VM Based systems 

- Renovo 

- Polyunpack 

- Zynamics Bochs unpacker 

Problems 

- Detection of virtual machines is easy 

- Intel CPU never traditionally designed for 
virtualization 

- Do not emulate x86 bug-for-bug 
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OS Integrated Monitoring 

Saffron, OllyBonE 

- Page-fault handler based debugger 

- Abuses the supervisor bit on memory pages 

- High-level executions per page 
Problems 

- Destabilizes the system 

- Need dedicated hardware 

- Fine-grain monitoring not possible 



NEW MEXICO TECH 

SCIENCE • ENGINEERING • RESEARCH • UNIVERSITY 




m 



Fully Hardware Virtualizations 

Ether: A. Dinaburg, P. Royal 

- Xen based hypervisor system 

- Base functions for monitoring 

• System calls 

• Instruction traces 

• Memory Writes 

- All interactions done by memory page mapping 
Problems 

- Old version of Xen hypervisor 

- Requires dedicated hardware 
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Disassembly and Code Analysis 

Most nebulous portion of the process 

Largely depends on intuition 

Looking at assembly is tedious 

Suffers from "not seeing the forest from the 
trees" syndrome 

Analyst fatigue - Level of attention required 
yields few results 
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Find Interesting and Relevant Portions 
of the Executable 

• Like disassembly, this relies on a lot of 
intuition and experience 

• Typical starting points: 

- Look for interesting strings 

- Look for API calls 

- Examine the interaction with the OS 

• This portion is fundamentally imprecise, 
tedious, and often frustrating for beginners 
and experts 
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Hypervisors 

• Lots of hype over the past few years 

• New hypervisor rootkits lead defensive tools 

• Covert methods for analyzing runtime 
behavior are extremely useful 

• Detection of hardware virtualization not 
widely implemented 
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Useful Hypervisor Technology 

VMWareESX Server 

- Commercial grade solution for VMs 

- Avoids VM detection issues (mostly) 
Linux Kernel Virtual Machines (KVM) 

- Separates analysis OS from target OS (slightly safer?) 

- Uses well-tested Linux algorithms for analysis 
Xen 

- Excellent set of tools for introspection 

- Uses standard QEMU image formats 

- API Controlled via Python - Integration into tools is 
easier 
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Contributions 

Modifications to Ether 

- Improve malware unpacking 

- Enable advanced tracing mechanisms 

- Automate much of the tedious portions 

Visualizing Execution for Reversing and 
Analysis (VERA) 

- Speed up disassembly and finding interesting 
portions of an executable 

- Faster identification of the Original Entry Point 
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What is Ether? 

Patches to the Xen Hypervisor 
Instruments a windows system 
Base modules available 

- Instruction tracing 

- API Tracing 

- Unpacking 

"Ether: Malware Analysis via Hardware 
Virtualization Extensions" 
Dinaburg, Royal, Sharif, Lee 

ACM CCS 2008 



NEW MEXICO TECH 

SCIENCE • ENGINEERING • RESEARCH • UNIVERSITY 




Ether Event Tracing 

Detects events on an instrumented system 

- System call execution 

- Instruction execution 

- Memory writes 

- Context switches 
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Instruction Tracing 

EFLAGS register modified for single-step 
(trap flag) 

PUSHF and POPF instructions are intercepted 

Modifications to this single-stepping 
effectively hidden (except 
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Memory and System Calls 

Memory Writes 

- Tracked by manipulating the shadow page table 

- Gives access to the written and read memory 
addresses 

System Calls 

- Modifies the SYSENTER_EIP register to point to 
non-paged address space 

- Logged, returned to ether 

- Overrides 0x2e interrupt to catch older syscalls 
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Ether System Architecture 




[ Ether Analysis System 
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Extensions to Ether 

• Removed unpacking code from hypervisor into 
user-space 

• Better user mode analysis 



• PE Repair system - Allows for disassembly of 
executables 



• Added enhanced monitoring system for 
executables 
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User mode Unpacking 

Watch for and monitor all memory writes 

Allow program to execute 

When execution occurs in written memory, dump 
memory 

Each dump is a candidate for the OEP 

Not perfect, but very close 

Scaffolding for future modifications 
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PE Repair 

Dumped PE files had problems 

- Sections were not file aligned 

- Address of Entry Point invalid 

- Would not load in IDA correctly 

Ported OllyDump code to Ether user mode 

- Fix section offsets to match data on disk 

- Repair resources as much as possible 

- Set AddressOfEntryPoint to be the candidate OEP 
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Results 

Close to a truly covert analysis system 

- Ether is nearly invisible 

- Still subject to bluepill detections 

Fine-grain resolution of program execution 

Application memory monitoring and full 
analysis capabilities 

Dumps from Ether can now be loaded in IDA 
Pro without modification 
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Ether Unpacking Demo! 
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Open Problems 

• Unpacking process produces lots of candidate 
dump files 

• Better Original Entry Point discovery method 

• Import rebuilding is still an issue 

• Now that there is a nice tool for tracing 
programs covertly, we need to do analysis 



NEW MEXICO TECH 

SCIENCE • ENGINEERING • RESEARCH • UNIVERSITY 




m 



Overview 
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• Hypervisors and You 
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Modifying the Process 

Knowing what to look for is often the portion 
that most new reversers have trouble with 

Having an idea of the execution flow of a 
program is extremely useful 

- IDA is focused on the function view 

- Extend to the basic block view 
Software armoring removal made easy 
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Visualization of Trace Data 

Goals: 

- Quickly visually subvert software armoring 

- Identify modules of the program 

• Initialization 

• Main loops 

• End of unpacking code 

- Figure out where the self-modifying code ends (OEP 
detection) 

- Discover dynamic runtime program behavior 

- Integrate with existing tools 



NEW MEXICO TECH 

SCIENCE • ENGINEERING • RESEARCH • UNIVERSITY 




m 



Visualizing the OEP Problem 

Each block (vertex) represents a basic block 
executed in the user mode code 

Each line represents a transition 

The thicker the line, the more it was executed 

Colors represent areas of memory execution 
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VERA 

Visualization of Executables for Reversing and 
Analysis 

Windows MFC Application 

Integrates with IDA Pro 

Fast, small memory footprint 
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Visualizing Packers 



Memory regions marked for PE heuristics 



Color Key: 

Normal 

Nd section present 

Section SizeOfRawData = 



Instruction not present in packed executable 
Operands don't match 
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Netbull Virus (Not Packed) 




Netbull Zoomed View 




Visualizing Packers 



Memory regions marked for PE heuristics 



Color Key: 

Normal 

Nd section present 

Section SizeOfRawData = 



Instruction not present in packed executable 
Operands don't match 
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UPX 



Color Key: 
Normal 

section present 

Section SizeOfRawData : 




lusti uction not present in picked exec 
Operands don't match 




UPX-OEP 












ASPack 



Color Key: 
Normal 

section present 
Section SizeOfRawData = D 



Instruction not present in picked exec 
Operands don't match 




Instruction not present in picked executable 
Operands don't match 



Color Key: 
Normal 

section present 
Section SizeOfRawData : 




Instruction not present in picked exe 

Operands don't match 



TeLock 



Color Key: 
Normal 
o section present 
Section SizeOfRawData = 




Instruction not present In picked exec 
Operands don't match 






User Study 

Students had just completed week long 
reverse engineering course 

Analyzed two packed samples of the Netbull 
Virus with UPX and MEW 

Asked to perform a series of tasks based on 
the typical reverse engineering process 

Asked about efficacy of visualization tool 
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User Study: Tasks Performed 

• Find the original entry point (OEP) of the 
packed samples 

• Execute the program to look for any 
identifying output 

• Identify portions of the executable: 

- Packer code 

- Initialization 

- Main loops 



NEW MEXICO TECH 

SCIENCE • ENGINEERING • RESEARCH • UNIVERSITY 




m 









Original Entry Point Recognition 
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Initialization Recognition 



Found Init 
(0=No; l=Yes) 

■ Init SI 

■ Init S2 




Main Loop(s) Recognition 



Found Loop(s) 
(0=No; 



Main Loops SI 
I Main Loops S2 




Main Loops S2 
Main Loops SI 



Overall Evaluation 



= No; 
l=Yes 



Likely to Use Again 
I Will Recommend 




Will Recommend 
Likely to Use Again 



Selected Comments 

"Wonderful way to visualize analysis and to 
better focus on areas of interest" 

"Fantastic tool. This has the potential to 
significantly reduce analysis time." 

"It rocks. Release ASAP." 
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Recommendations for improvement 

Need better way to identify beginning and end 
of loops 

Many loops overlap and become convoluted 

Be able to enter memory address and see 
basic blocks that match 
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Future Work 

General GUI / bug fixes 

Memory access visualization 

System call integration 

Function boundaries 

Interactivity with unpacking process 

Modify hypervisor to work with WinDBG, 
OllyDbg, IDA Debugger 
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Conclusions 

Visualizations make it easy to identify the OEP 

No statistical analysis of data needed 

Program phases readily identified 

Graphs are relatively simple 

Preliminary user study shows tool holds 
promise for speeding up reverse engineering 
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• Cort Dougan 




• Moses Schwartz 




• Alan Erickson 




• Alex Kent 




• New Mexico Tech SFS Program 
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Closing thoughts 

Ether is awesome. Thanks Artem Dinaburg 
and Paul Royal. 

Source, tools, and latest slides can be found 

at: 

http://www.offensivecomputing.net 

If you use the tool, please give feedback 

Look for the paper at Vizsec 2009 
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